Uncertainty in quantitative health impact modelling

Christopher Jackson
MRC Biostatistics Unit, University of Cambridge

Uncertainty in HIA models: Summary of lecture

  • Parametric models — definitions

  • Quantifying parameter uncertainty using probability distributions

  • Questions we can answer better by quantifying uncertainty

    • Uncertainty analysis — how confident are our results, how strong is the evidence?

    • Sensitivity analysis — how would results change if part of the model changed?

    • Value of Information (VoI) analysis — what parts of the model do we need better evidence for?

Quantitative models

Models approximate process that generates some output of interest, helping to inform decision-making.

Example: ITHIM health impact model [ TODO borrow picture ]

  • inputs: travel data, physical activity, air pollution, traffic injuries
    scenarios of change in these

  • outputs: population health (and changes under scenarios)

These generally involve parameters. What are these?

Parametric models

Model input parameters: examples

  • background exposure, e.g. average levels of an air pollutant, or levels of physical activity (for some population, area, characteristics)

  • relative risk of getting a disease, given some change in some exposure

Parameters are representations of knowledge about a (real or imagined) population.

  • Usually conceive as summaries of individual quantities over a large/infinite “population”.

Model transform parameters \(\rightarrow\) output quantities.
Outputs also represent summaries over populations

Exercise / discussion 1

What is a parameter and why are parameters uncertain (5 min)

Parameters are generally uncertain

Two broad reasons for parameter uncertainty: the parameter estimate is

  • an observed summary of a limited population, and/or

  • from a population that is different from the one we want

Models are also uncertain (“structural” uncertainty)

  • By definition, model approximates reality (“all models are wrong, but some are useful”…)

  • Ideally quantify as many uncertainties as possible inside the model

Individual variability versus uncertainty about knowledge

Individual-level quantities Parameters
individual exposure to PM2.5 during a trip expected exposure to PM2.5 for a trip/individual of this kind
whether this individual dies within a year risk of death within a year

Individual-level quantities are different for each individual

Our knowledge of parameters may be uncertain

Individual variability can never be removed, but parameter uncertainty can be reduced with better knowledge

Why does uncertainty matter?

We’ve built best model we can. Just report our “best estimate” to the decision maker?

  • Decision makers need good evidence to change practice — models should indicate strength of evidence for result

  • What about the future - we may be able to get better evidence - what research should be done?

Here we will cover quantitative methods for assessing uncertainty. In particular, probabilistic methods.

  • though there is a broader field of appraising evidence, engaging with stakeholders or experts etc…

Questions relevant to uncertainty quantification

Uncertainty analysis: about strength of evidence

  • What range of outputs are plausible, given current evidence?

  • Which parameters most influence the output uncertainty?

Sensitivity analysis: “what if…”

  • What if parameter took the value \(b\) instead of \(a\) — how would the output change?

Value of Information analysis

  • How much would the model improve if we got better information?
    Which parameter should we get better information on?

Uncertainty in microsimulation models

(advanced, may remove this slide)

Microsimulation models create a population of synthetic individuals and simulating / summarising quantities of interest

Often based on drawing individuals directly from an observed dataset

  • (in contrast with models where inputs and outputs represent population summaries)

ITHIM: synthetic population of “individuals” (households, people, trips) based on travel survey data. Exposures assigned to trips. Health impacts for aggregated age groups.

Uncertainty here applies to individual-level quantities (e.g. trips taken for a person with given characteristics), and is represented by variability between individuals in the survey data. This approach doesn’t consider uncertainty about quality/relevance of the travel survey dataset

How to quantify uncertainty

Two broad approaches

  1. Statistical analyses of data (your own, or published analyses) giving point and interval estimates or standard errors

  2. Judgements (informal or from structured expert elicitation), e.g.

  • point estimate (best guess) for parameter value

  • credible interval e.g. “I judge that the parameter is between \(a\) and \(b\), with 95% confidence”

Our goal in each case is to obtain probability distributions for parameters

  • rigorous basis for uncertainty and sensitivity analysis

Probability distributions

A full probability distribution

For any pair of values \((a,b)\), we can deduce the probability that the parameter is between \(a\) and \(b\).

Statistical analyses: Bayesian methods

In statistical models, data assumed to come from models with parameters…

e.g. number of disease cases is Binomial (population n, underlying prevalence p)

  • Bayesian methods based around quantifying parameter uncertainties as probability distributions

  • Prior distribution combined with study data \(\rightarrow\) posterior distribution.

Prior distribution dominates if data are weak
Prior doesn’t matter if enough data

Statistical analyses: frequentist methods

  • Construct an estimator of parameters based on a finite dataset

  • Uncertainty quantified by imagining different datasets drawn from the same population

    • Standard error: variability in estimates between datasets.

    • Confidence interval: contains true value in 95% of datasets drawn

If dataset is large, can interpret a frequentist analysis as Bayesian: parameter has a normal distribution, defined by estimate and standard error

If dataset is small, Bayesian analyses have the benefit of allowing background information to be included as a prior

Examples of statistical models that inform quantitative HIA

Global

  • Epidemiological models for estimating disease incidence and mortality

  • Meta-analysis of relative risks (of disease outcomes given different exposures)

Local

  • Road traffic deaths or injuries: count data (e.g. Poisson) regression models of reported incidents by person/vehicle characteristics

  • Models for analysing travel survey data

Examples of judgements: physical activity

MMET value for cycle commuting. Data from Compendium of Physical Activities (https://pacompendium.com/bicycling/)

Which of these are relevant to our particular health impact model?

Examples of judgements: physical activity

Might judge that code 1011 (MET 6.8) is a typical kind of cycling for our context, and judge a credible interval based on similar kinds of cycling in this table.

Rough judgement: no “correct answer”!

Note: Does the parameter in our model represent the average for some population, or the value for a person / trip?

Note also: Each published MMET value is itself an estimate, that conceals variability and uncertainty!

Examples of judgements: air pollution

PM2.5 concentration. We know published estimates of average and SD for 30 cities in India.

Suppose our city is not one of these. We just know the average PM2.5 for our city. Where do we get a credible interval or SD?

Indian data shows a “typical” set of SDs of pollution within cities. So we might use an average of these SDs. But…

  • What do we know about how our city compares to these?

  • Do we want a measure of uncertainty (about an average) or variability (between points)?

Source: https://urbanemissions.info/wp-content/uploads/apna/docs/2019-07-APnA30city_summary_report.pdf

Meta-analysis

Formally averaging a quantity estimated from different studies.

Give more weight to more confident studies and/or those closer to our context.

Most developed in randomised clinical trials — highly controlled, regulated studies

see e.g. https://training.cochrane.org/interactivelearning for learning resources — find those about observational studies in epidemiology

Might summarise the average of the observed studies, or make a prediction for a new study (more uncertain)

If studies disagree, consider why. Ensure relevant studies selected.

Considerations for uncertainty quantification

Not always possible to precisely quantify uncertainty

  • Rough, clearly-stated judgement better than ignoring it

  • Uncertainty about a parameter may or may not affect the uncertainty about a model output (see “Value of Information”, later…)

    • If output is robust to uncertainty, conclusion is strengthened

When making judgements, consider, e.g.:

  • quality/amount of data behind published numbers. how were these numbers obtained?

  • relevance of population, variations over time / between places

Exercise / discussion 2

Quantifying judgements with probability (5 min)

Distributions from credible intervals

Suppose we have a parameter with estimate 0, credible interval (-2 to 2)

How to derive a probability distribution that reflects this belief?

What is wrong with a distribution like this

Distributions from credible intervals

Suppose we have a parameter with estimate 0, credible interval (-2 to 2)

A triangular distribution is a bit more plausible

Distributions from credible intervals

Suppose we have a parameter with estimate 0, credible interval (-2 to 2)

A normal distribution is even better

Normal distribution

Used for quantities with unrestricted ranges

Defined by mean \(\mu\) and standard deviation \(\sigma\) (or variance \(\sigma^2\))

95% credible interval is \(\pm 2\) SDs: i.e. width is 4 SDs.: SD easily derived from a CI.

Log-normal distribution

Used for positive-valued quantities.

Normal distribution for the log of the quantity

Example: MMET/h estimate 2.5 (CI 1 to 4).

  • Transform to log(MMET).

  • Estimate log(2.5), CI (log(1) to log(4)).

  • Assign normal with SD = CI width / 4

If the SD is published, but not the CI
e.g. MMET/h estimate \(m=2.5\) (SD \(s=1.4\))?

Method of moments gets us mean \(\mu\) and SD \(\sigma\) on log scale

\(\mu = \log(m/\sqrt{s^2/m^2 + 1}), \sigma= \sqrt{\log(s^2/m^2 + 1)}\)

Beta distribution

Used for quantities between 0 and 1: probabilities, proportions

Defined by “shape” parameters \(a,b\)

Given an estimate \(m\) and credible interval, how to obtain \(a, b\)?

Approximate SD \(s =\) CI width/4, then use “method of moments”:
\(a = (m(1-m)/s^2 - 1)m\)
\(b = (m(1-m)/s^2 - 1)(1 - m)\)

“SD = CI width / 4” heuristic based on symmetric, normal-like distributions.
Less accurate in example (b) opposite.
Could adjust by trial and error to match desired belief more closely.

See SHELF R package for more sophisticated techniques for fitting distributions

Exercise 3

Obtaining full probability distributions from published estimates and uncertainties (20 min)

Doing uncertainty analysis: Monte Carlo simulation

For each \(i = 1, 2, \ldots N\) (enough to give precise summaries)

  1. Simulate parameters \(X_i\) from their uncertainty distributions

  2. Compute the model output \(Y_i = g(X_i)\)

producing a sample from the model outputs \(Y_1, \ldots, Y_N\) [ animate? ]

Summarise the sample to give e.g. 

  • a credible interval for the outputs

  • probability that e.g. number of deaths \(>\) [important value]

How many samples to run?

As \(N\) gets greater, the summaries of the sample will not change much when you run more.

  • run enough until you have the precision you want

  • do you really need more than 2 or 3 significant figures?

More precisely: the Monte Carlo standard error (MCSE) describes how accurate a summary of a Monte Carlo sample is.

  • For the sample mean, the MCSE is \(SD(sample) / \sqrt{N}\)

  • This reduces to zero as \(N\) gets larger.

  • We can then say the mean of the sample is precise to \(\pm\) 2 MCSE

If your model is very slow and doing Monte Carlo is too expensive to get the precision you want, then present MCSE alongside the estimate to show your readers how much precision you do have.

Technical point - uncertainty affects “best estimate” in nonlinear models

Given a model with uncertain inputs \(X\), with expected values \(E(X)\)
Outputs \(Y = g(X)\) where \(g()\) is function that defines the model

What is \(E(Y)\), the expected value of \(Y\)?

Can we just plug in the best estimates of the inputs, as \(g(E(X))\)?

\(E(Y)\) only equals \(g(E(X))\) if the function \(g()\) is linear:

Proof

\(g(E(x)) = g((X_1 + \ldots + X_n)/n)\), then if \(g()\) is linear, this equals
\((g(X_1) + \ldots + g(X_n)) / n = E(g(X)) = E(Y)\).

Most realistic models are non-linear, therefore need Monte Carlo simulation to get the true expectation of the output.

One-way sensitivity analysis

Answers questions like “what if [parameter] was actually \(b\) instead of \(a\)”

For each parameter, compare model output with

  • parameter at a “low value”

  • parameter at a “high value”

What if all the other parameters are uncertain? Probabilistic one-way sensitivity analysis:

  • Fix [parameter] at high or low value

  • Run the model under Monte Carlo analysis, using the uncertainty distributions of the other parameters

Tornado plot

Limitations of one-way sensitivity analysis

Arbitrary choice of “low” and “high” value may not convey the effect of the parameter

e.g. if effect of parameter is nonlinear

What if parameters have correlated effect on the outcome?
Hard to vary more than one parameter at a time

Exercise 5

Monte Carlo simulation of health impact models (30 min)

Value of Information

Recall

  • sensitivity analysis: “what if model was a bit different”

  • uncertainty analysis: “what is strength of evidence in model”

A different (related) question: “what would be the benefit of getting better information”.

This is Value of Information analysis

Value of Information

Given current information, model output is uncertain. But how much more precise would it get…

if we were to learn some parameter exactly?

  • Expected value of partial perfect information

if we conducted a study (say, a survey of 100 people) to estimate it

  • Expected value of sample information

Helps us to:
set research priorities to reduce uncertainty
design studies (more advanced, not covered here)

Value of Information in terms of health economics

Value of Information methods developed in health economics

  • Model a health policy decision and its consequences (e.g. health benefits as QALYs vs costs). Parameter uncertainties as probability distributions

  • Information has value: \(\rightarrow\) reduces parameter uncertainty \(\rightarrow\) more precise model outputs \(\rightarrow\) better informed policy-making \(\rightarrow\) health benefits

Not previously used much in health impact modelling, where models used more for scenarios than policies

How can we define “value” if we don’t model a health policy?

Precision as value

Define value as “precision of estimate” (e.g. health impact of scenario)

  • How much will the variance (or SD, or credible interval width) be expected to reduce if we got better information?

  • More precise estimates (implicitly) assumed to ultimately lead to benefits

  • If needed, trade off informally with costs of research (willingness to pay for more precise estimates? Not covered here)

How exactly to do these computations?

Computing expected value of partial perfect information (EVPPI)

Model \(Y = g(X_1, X_2,.. )\) with some (scalar) output \(Y\) and multiple inputs \(X_r\)

Do uncertainty analysis: define distributions for each \(X_r\), obtain distribution for \(Y\) via Monte Carlo simulation.

\(var(Y)\): variance of model output under current information

Definition of EVPPI - expected reduction in this variance if we were to learn the exact value of \(X_r\) (say)

\[var(Y) - E_x(var(Y | X_r = x))\]

We don’t know the value of \(X_r\) when we calculate this — so we must take the expectation over possible values \(x\) (using the distribution we defined)

Illustration of EVPPI computation

Take the Monte Carlo sample, with all parameters uncertain

Regression of \(Y\) = model output versus \(X\) = parameter of interest

Expected reduction in variance of \(Y\) if we learnt \(X\)

\[var(Y) - E_x(var(Y | X = x))\]

We don’t know \(X\) so we average over our uncertainty

\(var(Y)\): uncertainty before learning \(X\)

\(E_x(var(Y | X = x))\): residual variance: mean of (“observed” - fitted) over different \(x\)

Spline regression for computing EVPPI

Regression function of output on input will not necessarily be linear.

Spline regression (automated choice of smoothness, works well enough in this context)

lm(Y ~ X)  # linear model
library(mgcv)
gam(Y ~ X) # spline ("generalized additive") model

The voi package can take care of extracting ingredients needed for the EVPPI computation (see the practical)

library(voi)
evppivar(outputs, inputs)

EVPPI for more than one parameter at once

Example: dose-response curve governed by four different parameters \(\alpha\), \(\beta\), \(\gamma\), \(\tau\). Estimate the expected value of jointly learning all four of them (hence learning the dose-response curve)

Regression model with four predictors, e.g.

gam(Y ~ alpha + beta + gamma + tau)

More advanced regression models available, which automatically choose the best fitting regression function for Y given the predictors (and their interactions): see the practical about the voi package

Presenting EVPPI results

Impact of different parameters on the output uncertainty, presented as

Uncertainty that would remain in the output \(Y\) if we knew \(X\)

  • as a variance, \(var(Y) - EVPPI_X\)

  • …or standard deviation (square root of this)

  • …or rough credible interval (\(\pm 2\) remaining standard deviation)

or proportion of uncertainty (variance) in \(Y\) explained by \(X\)

Value of Information: further topics

We will never get perfect information, but we may get better information via a new research study.

Expected value of sample information is the expected value of a study of a particular design / sample size

  • Slightly harder to define and compute

  • Previously used in health economic decision models, but not (yet!) used in health impact models

  • See voi package and references there for some examples.

Value of Information: further topics

Here we have just talked about “value” as variance of outputs

Value of Information also used widely for health economic decision models

  • Model chooses policy with the highest net benefit (or lowest loss)

  • “Value” is the net benefit (NB) itself

  • Value of information is NB(better information) - NB(current information)

See voi package and references there for some examples.

Practical session 5

Value of Information analysis, using the voi package

(Worked example, 20 min)

Discussion

What uncertainty/sensitivity questions do you want to answer in your own work?

Do you have the tools and skills to do this, after this course?